Image pick-up apparatus and control method

ABSTRACT

An image pick-up apparatus detects shake and the like applied to an image pick-up apparatus by a vibration sensor. A motion vector detection unit detects a motion vector of an image in an image signal by an imaging unit. A feature point tracking unit calculates coordinate values of a subject on an imaging screen that changes over time on the basis of the motion vector. A feature coordinate map and a position and attitude estimation unit estimates a position and attitude of the image pick-up apparatus and a positional relationship including a depth between the subject and the image pick-up apparatus based on an output of the vibration sensor and the coordinate values of the subject. A computation unit calculates a control amount of image blur correction using feature points of a main subject, a feature coordinate map and position or attitude information of the image pick-up apparatus. A correction lens is driven according to an output of a target position calculation unit and a shake correction operation of the image pick-up apparatus is performed.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image blur correction technology foran optical device such as a video camera, a digital still camera, and aninterchangeable lens thereof.

Description of the Related Art

An optical image blur correction process and an electronic image blurcorrection process are provided as functions for correcting image blurof a subject that is generated due to hand shake or the like of a userwho holds a main body part of an image pick-up apparatus. In the opticalimage blur correction process, a vibration of the main body part isdetected by an angular velocity sensor or the like, and control isperformed so that a correction lens provided in an imaging opticalsystem is moved according to the detection result. When an optical axisdirection of the imaging optical system is changed, and an image that isformed on a light receiving surface of an image pickup element is moved,image blur can be corrected. In addition, in the electronic image blurcorrection process, image blur is artificially corrected by performingimage processing on a captured image.

Photographers have a desire to perform imaging while moving with asubject (a moving object or a stationary object) and keeping the subjectwithin an angle of imaging view. An operation of tracking a selectedsubject so that a detection position of a subject form is close aspecific position in an imaging screen may be referred to as a “subjecttracking operation.” A movement intended by the photographer at thattime is referred to as “camera work.” For example, movement of an imagepick-up apparatus by a user to bring a detected subject position to (ornear) a specific position in the apparatus's imaging screen may bereferred to as camera work. The terms “subject tracking operation” and“camera work” are used throughout the application. The specific positionmay be, for example, a center position of the imaging screen or aposition designated by the photographer. In addition, there is a methodof assisting the subject tracking operation by an image blur correctionunit. In Japanese Patent Laid-Open No. 2010-93362, a subject trackingtechnology for driving an image blur correction unit in which the insideof a screen is divided into blocks, the face or the like of a subject isdetected by template matching, and a movement of the subject is trackedand set to be within the screen is disclosed.

On the other hand, in order to correct image blur due to hand shake orthe like, it is necessary to detect a change in the position andattitude of the image pick-up apparatus. As a self-position estimationmethod for detecting an attitude and position of the image pick-upapparatus, a position and attitude estimation (visual and inertialsensor fusion) technology using structure from motion (SFM) and aninertial sensor is provided. A method of estimating a 3D position of anobject present in a real space and a position and attitude of an imagepick-up apparatus by applying this technology is known.

In the method disclosed in Japanese Patent Laid-Open No. 2010-93362, ashake correction operation is performed on the basis of change in theposition and attitude of the image pick-up apparatus in which camerawork that is intentionally performed by a photographer for subjecttracking and change in position and attitude due to hand shake coexist.However, an issue with present shake correction operations is that theymay cancel out changes in position and/or attitude of an image pick-upapparatus that arise from camera work. This is of course undesirablebecause camera work by a photographer is essentially a desired movementfor following a subject with the image pick-up apparatus. As a result,present shake correction operations may lead to an unnatural change inangle of view in a captured image.

SUMMARY OF THE INVENTION

The present invention reduces an unnatural change in angle of view dueto image blur correction in imaging according to camera work.

An apparatus according to an embodiment of the present invention is animage pick-up apparatus that acquires an image signal by an imagingunit, the apparatus including a memory; and one or more processors,wherein the processor functions as the following units according to aprogram stored in the memory: a first acquisition unit configured toacquire first information indicating shake of the image pick-upapparatus detected by a shake detection unit; a second acquisition unitconfigured to acquire second information indicating a movement of asubject detected in an image signal by the imaging unit; a tracking unitconfigured to calculate coordinate values of the subject on an imagingscreen using the second information and track feature points; anestimation unit configured to estimate a position and/or attitude of theimage pick-up apparatus and a positional relationship including a depthbetween the subject and the image pick-up apparatus from the firstinformation and the coordinate values of the subject; a computation unitconfigured to calculate a control amount of shake correction using (i)the estimation value of the position or attitude of the image pick-upapparatus acquired from the estimation unit, (ii) the positionalrelationship acquired from the estimation unit, (iii) the firstinformation and (iv) the calculated coordinate values of the subject;and a correction unit configured to correct image blur due to shake ofthe image pick-up apparatus based on the control amount calculated bythe computation unit.

Further features, advantages and aspects of the present invention willbecome apparent from the following description of exemplary embodiments(with reference to the attached drawings). It should be understood thatany of the features described herein in relation to a particularembodiment or set of embodiments may be combined with the features ofone or more other embodiments without any limitations other than thoseimparted by the broadest aspects of the invention as definedhereinabove. In particular, features from different embodiments can becombined where necessary or where the combination of elements orfeatures from individual embodiments in a single embodiment isbeneficial.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing a configuration example of an image pick-upapparatus according to an embodiment of the present invention.

FIG. 2 is a diagram showing a configuration example of an image blurcorrection device according to a first embodiment of the presentinvention.

FIG. 3A and FIG. 3B show diagrams of a configuration of a targetposition calculation unit, and a main subject feedback amountcomputation unit.

FIG. 4 is a flowchart of a target position calculation process accordingto the first embodiment.

FIG. 5 is a flowchart of a position and attitude estimation processaccording to the first embodiment.

FIG. 6 is a flowchart of a main subject feedback amount computingprocess according to the first embodiment.

FIG. 7 is a diagram showing the relationship between a coordinateposition of an object in world coordinates and a coordinate position incamera coordinates.

FIG. 8 is a diagram showing a perspective projection model in which avirtual imaging plane is set at a position in front of a lens.

FIG. 9A and FIG. 9B are diagrams showing position and attituderelationships between a main subject, a background subject close to themain subject, and an image pick-up apparatus.

FIG. 10A and FIG. 10B are diagrams showing the relationship betweenmovements of feature points of a main subject and a background in animaging operation.

FIG. 11 is a diagram showing a configuration example of an image blurcorrection device according to a second embodiment of the presentinvention.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described with reference tothe appended drawings. In the embodiments, an image blur correctiondevice configured to perform image blur correction on a captured imageis exemplified. The image blur correction device is configured to driveand control a movable member and the like of an image blur correctionoptical system. The image blur correction device and/or the image blurcorrection optical system can be mounted in an image pick-up apparatussuch as a video camera, a digital camera, a silver salt still camera andan optical device (e.g. an observation device such as binoculars, atelescope, and a field scope). In addition, the image blur correctiondevice can be mounted in an optical device such as an interchangeablelens for a digital single-lens reflex camera. An operation of performingimage blur correction using a shake detection signal of a device will bereferred to below as an “image blur correction operation.”

First Embodiment

FIG. 1 is a block diagram showing a configuration example of an imagepick-up apparatus according to the present embodiment. The image pick-upapparatus 100 is, for example, a digital still camera, and has a movingimage capturing function.

An image pick-up apparatus 100 includes a zoom unit 101. The zoom unit101 constitutes an image forming optical system and includes a zoom lensby which an imaging magnification is changed. A zoom drive unit 102drives the zoom unit 101 according to a control signal from a controlunit 119. An image blur correction lens (hereinafter referred to as acorrection lens) 103 is a movable optical member that can be moved tocorrect image blur. The correction lens 103 is movable in a directionorthogonal to an optical axis of an imaging optical system. An imageblur correction lens drive unit 104 controls driving of the correctionlens 103 according to a control signal from the control unit 119. Anaperture and shutter unit 105 includes a mechanical shutter having anaperture function. An aperture and shutter drive unit 106 drives theaperture and shutter unit 105 according to a control signal from thecontrol unit 119. A focus lens 107 is a movable lens that is used forfocus adjustment, and has a position that can be changed along theoptical axis of the imaging optical system. A focus drive unit 108drives the focus lens 107 according to a control signal from the controlunit 119.

The imaging optical system forms an image on an image unit 109. An imagepickup element (such as a CCD image sensor and a CMOS image sensor) ofthe imaging unit 109 converts the optical image into an electricalsignal representative of the image. The electrical representation of theimage may be formed of pixels. The term CCD is an abbreviation of a“charge coupled device.” The term CMOS is an abbreviation of a“complementary metal-oxide semiconductor.” An imaging signal processingunit 110 performs analog/digital (A/D) conversion, correlated doublesampling, gamma correction, white balance correction, colorinterpolation processing, and the like on the electrical signal outputfrom the imaging unit 109, and converts it into a video signal.

A video signal processing unit 111 processes the video signal acquiredfrom the imaging signal processing unit 110 according to variousapplications. Specifically, the video signal processing unit 111generates a video signal for display and performs an encoding processand a data file conversion process for recording. A display unit 112performs image display as necessary according to a video signal fordisplay output from the video signal processing unit 111. A power supplyunit 115 supplies power to respective units of the image pick-upapparatus 100 according to applications. An external input and outputterminal unit 116 is used to input and output a communication signal anda video signal to and from an external device. An operation unit 117includes operation members such as buttons and switches for a user toissue an instruction to the image pick-up apparatus 100. For example,the operation unit 117 may include a release switch configured tosequentially turn a first switch (denoted as an SW1) and a second switch(denoted as an SW2) on according to an amount pushing of a releasebutton. In addition, the operation unit 117 includes switches forsetting various modes. A storage unit 118 stores various types of dataincluding video information and the like.

The control unit 119 includes, for example, a CPU, a ROM, and a RAM. TheCPU is an abbreviation of a “central processing unit.” The ROM is anabbreviation of a “read only memory.” The RAM is an abbreviation of a“random access memory.” The CPU loads a control program stored in theROM into the RAM and executes it, controls respective units of the imagepick-up apparatus 100, and realizes various operations to be describedbelow. When the SW1 is turned on by a half-pressing operation of therelease button included in the operation unit 117, the control unit 119calculates an auto focus (AF) evaluation value on the basis of a videosignal for display that is output from the video signal processing unit111 to the display unit 112. The control unit 119 controls the focusdrive unit 108 on the basis of the AF evaluation value and therebyperforms automatic focus detection and focus adjustment control. Inaddition, the control unit 119 performs automatic exposure (AE)processing for determining an aperture value and a shutter speed forobtaining an appropriate exposure amount on the basis of video signalluminance information and a predetermined program line diagram. Inaddition, when the SW2 is turned on by a full-pressing operation of therelease button, the control unit 119 performs an imaging process withthe determined aperture value and shutter speed, and controls respectiveprocessing units so that image data obtained by the imaging unit 109 isstored in the storage unit 118.

The operation unit 117 includes an operation switch that is used toselect an image blur correction (stabilizing) mode. When an image blurcorrection mode is selected by an operation of the operation switch, thecontrol unit 119 instructs the image blur correction lens drive unit 104to perform an image blur correction operation. The image blur correctionlens drive unit 104 performs an image blur correction operationaccording to a control instruction of the control unit 119 until aninstruction of turning image blur correction off is issued. In addition,the operation unit 117 includes an imaging mode selection switch thatcan select either a still image imaging mode or a moving image imagingmode. A process of selecting an imaging mode is performed by a useroperation of the imaging mode selection switch, and the control unit 119changes operation conditions of the image blur correction lens driveunit 104. The image blur correction lens drive unit 104 constitutes theimage blur correction device of the present embodiment. In addition, theoperation unit 117 includes a reproduction mode selection switch forselecting a reproduction mode. When a user selects a reproduction modeby an operation of the reproduction mode selection switch, the controlunit 119 performs control so that the image blur correction operation isstopped. In addition, the operation unit 117 includes a magnificationchange switch for instructing zoom magnification change. When zoommagnification change is instructed according to a user operation of amagnification change switch, the zoom drive unit 102 that has receivedthe instruction from the control unit 119 drives the zoom unit 101 andmoves the zoom lens to the instructed position.

FIG. 2 is a diagram showing a configuration example of the image blurcorrection device of the present embodiment. A process of calculating adrive direction and a drive amount regarding the correction lens 103 andposition control will be described below. The image blur correctiondevice of the present embodiment includes a first vibration sensor 201and a second vibration sensor 203. The first vibration sensor 201 is,for example, an angular velocity sensor. The first vibration sensor 201detects vibration components (angular velocity) in a vertical direction(pitch direction), a horizontal direction (yaw direction), and arotation direction (roll direction) around an optical axis of the imagepick-up apparatus 100 in a general attitude (an attitude in which a longside direction of an imaging screen substantially matches a horizontaldirection). The first vibration sensor 201 outputs a detection signal toan A/D converter 202. The second vibration sensor 203 is, for example,an acceleration sensor. The second vibration sensor 203 detects anacceleration component in a vertical direction, an accelerationcomponent in a horizontal direction, and an acceleration component in anoptical axis direction of the image pick-up apparatus 100 in a generalattitude, and the second vibration sensor 203 outputs a detection signalto an A/D converter 204. The A/D converters 202 and 204 acquiredetection signals from the first and second vibration sensors, andconvert analog values to digital values. Here, while a case in which theshake detection unit includes the first vibration sensor and the secondvibration sensor has been exemplified in the present embodiment, thepresent invention can be applied to an embodiment in which the firstvibration sensor or the second vibration sensor is included.

A position detection sensor 212 detects a position of the correctionlens 103 and outputs a position detection signal to an A/D converter218. The A/D converter 218 acquires a detection signal from the positiondetection sensor 212 and converts analog values into digital values.

A target position calculation unit 213 calculates a control targetposition of the correction lens 103 based on outputs from the A/Dconverter 202 and a computation unit 219 to be described below. Thetarget position calculation unit 213 outputs a corrected positioncontrol signal of the correction lens 103 in a pitch direction and a yawdirection to a subtractor 214. The subtractor 214 subtracts a positiondetection signal from a corrected position control signal. The positiondetection signal may be received from the position detection sensor 212via the A/D converter 218, and the corrected position control signal isreceived from the target position calculation unit 213. The output fromthe subtractor 214 is provided to a control filter 215. The controlfilter 215 acquires the corrected position control signal from thetarget position calculation unit 213, together with a deviation ofposition information of the correction lens 103 from the positiondetection sensor 212, and performs feedback control. That is, thecontrol filter 215 outputs a control signal for image blur correction tothe image blur correction lens drive unit 104 (which has an actuator)and performs drive control of the correction lens 103.

Next, a drive control operation of the correction lens 103 by the imageblur correction device will be described in detail.

The target position calculation unit 213 (which may be referred toherein as an acquisition unit) acquires a shake detection signal(angular velocity signal) from the first vibration sensor 201 and a mainsubject feedback amount from the computation unit 219, and generates acorrected position control signal for driving the correction lens 103 ina pitch direction and a yaw direction. The corrected position controlsignal is output to the control filter 215 through the subtractor 214.

The position detection sensor 212 detects a position of the correctionlens 103 in a pitch direction and a yaw direction, and outputs aposition detection signal to the control filter 215 through (i.e. via)the A/D converter 218 and the subtractor 214. The subtractor 214 outputsa signal obtained by subtracting the position detection signal from thecorrected position control signal to the control filter 215. The controlfilter 215 performs feedback control through the image blur correctionlens drive unit 104 so that a position detection signal value convergesto a value of the corrected position control signal from the targetposition calculation unit 213. The corrected position control signaloutput from the target position calculation unit 213 is a control signalfor moving the correction lens 103 so that image blur of a subject iscanceled out. For example, the target position calculation unit 213performs filter processing or the like on shake detection informationand generates a correction speed control signal or a corrected positioncontrol signal. When vibration such as hand shake is applied to theimage pick-up apparatus during imaging, image blur can be reduced up toa certain extent of vibration according to a control operation of movingthe correction lens 103.

FIG. 3A is a block diagram showing a detailed internal configuration ofthe target position calculation unit 213. A high pass filter 301performs a process of removing a direct current (DC) offset component ofa detection signal from the first vibration sensor 201. A low passfilter 302 performs a process of converting an angular velocity signalinto a signal corresponding to an angle. An integration gain unit 303multiplies an output of the low pass filter 302 by a predeterminedintegration gain. An adder 304 adds an output of the integration gainunit 303 and a main subject feedback amount. A main subject feedbackamount which is a control amount of shake correction will be describedbelow.

A target position calculation process will be described with referenceto FIG. 4. FIG. 4 is a flowchart showing a flow of the target positioncalculation process. The target position calculation unit 213 acquiresdata of a shake angular velocity of the image pick-up apparatus 100detected by the first vibration sensor 201 (S116). The high pass filter301 removes a DC offset component from the acquired data (S117). Inaddition, filter processing is performed by the low pass filter 302(S118), the integration gain unit 303 multiplies by a gain and anangular velocity signal of a shake component is convened into an anglesignal (S119). The adder 304 adds the main subject feedback amountcalculated by the computation unit 219 and an output of the integrationgain unit 303 (S120). The addition result is output to the subtractor214.

Next, a configuration in which a motion vector is detected in a capturedimage and feature points are tracked, and a position and attitude of theimage pick-up apparatus are estimated will be described with referenceto FIG. 2. Image data acquired by the imaging unit 109 is processed bythe imaging signal processing unit 110. A motion vector detection unit211 (which may be referred to herein as an acquisition unit) detects amotion vector of the captured image in the signal output from theimaging signal processing unit 110. A global vector computation unit 220calculates a global vector representing a uniform movement of the entireimaging screen based on detected motion vector information. A globalvector is calculated using a motion vector having the highest frequencyof occurrence, and global vector information is transmitted to the mainsubject feedback amount computation unit 219. A feature point trackingunit 209 performs a process of detecting and tracking a predeterminedfeature point in the captured image based on the detected motion vectorinformation. Herein, the feature point tracking unit 209 may simply bereferred to as a tracking unit.

A main subject separation unit 208 acquires an output of the featurepoint tracking unit 209 and specifies a coordinate area of a mainsubject in the captured image. The main subject is an important subject,and is determined by an image size, features of a subject (e.g. the faceof a person), an operation by a photographer, and the like. The mainsubject separation unit 208 extracts feature points of the main subjectcorresponding to tracking feature points, and separates a movement ofthe other feature points (such as the background). The main subjectseparation unit 208 outputs coordinates of feature points of the mainsubject to the computation unit 219, and outputs coordinates of featurepoints of the background other than the main subject to a featurecoordinate map and position and attitude estimation unit 205.

The feature coordinate map and position and attitude estimation unit 205estimates a position and attitude of the image pick-up apparatus 100 anda position of a feature point in a real space in which the image pick-upapparatus 100 captures an image using SFM and inertial sensorinformation. The feature coordinate map and position and attitudeestimation unit (hereinafter simply referred to as an estimation unit)205 includes a feature coordinate map estimation unit 206 and a positionand attitude estimation unit 207. An estimation process performed by thefeature coordinate map estimation unit 206 and the position and attitudeestimation unit 207 will be described below in detail.

A position and attitude estimation process performed by the estimationunit 205 will be described with reference to a flowchart in FIG. 5. Theprocesses of S101 to S105 and the processes of S108 to S115 areperformed in parallel. First, the processes of S108 to S115 will bedescribed.

The imaging unit 109 photoelectrically converts an optical signal formedthrough the imaging optical system into an electrical signal andacquires an analog image signal (S108). Next, the imaging signalprocessing unit 110 converts the analog image signal acquired from theimaging unit 109 into a digital image signal and performs predeterminedimage processing. The motion vector detection unit 211 detects a motionvector on the basis of the image signal (S109). When the motion vectoris detected, an image signal one frame ago stored in a memory in advanceis acquired (S112). This image signal and an image signal of the currentframe are compared, and a motion vector is calculated from adisplacement of the image. A method of detecting a motion vectorincludes a correlation method and a block matching method. A method ofcalculating a motion vector in the present invention is arbitrary.

The global vector computation unit 220 calculates a global vector fromthe detected motion vector information of the image (S110). When amotion vector value having the highest frequency of occurrence in thecaptured image is calculated in a known histogram process or the like, aglobal vector is calculated. The feature point tracking unit 209 detectsand tracks a movement position of a predetermined feature point in thecaptured image in coordinates of each frame when a moving image iscaptured (S111). Regarding a feature point tracking technology, there isa method in which a square window is provided around a feature point,and when a new frame of a target video is provided, a point having thesmallest residual in the window between frames is obtained. A trackingprocess may be performed using a known method, and details thereof willnot be described.

The main subject separation unit 208 specifies a coordinate area of themain subject in the captured image, extracts feature points of the mainsubject corresponding to tracking feature points, and separates movementof the other feature points (S113). Here, an area other than the mainsubject is set as a background area. As a subject detection method, forexample, there is a method in which color information is acquired froman image signal, histograms thereof are divided into mountain-likedistribution ranges, and partitioned areas are classified as onesubject, and the subject is detected. According to classification into aplurality of areas having similar image signals, it is possible todistinguish and detect a plurality of subjects. In addition, in movingimage capturing, in each imaging frame, there are feature points thatare continuously present in the captured image, and a feature point ofwhich a movement amount is different from other detected feature pointsand which has a smaller movement amount than the others is determined asa main subject. On the other hand, a feature point that disappears(leaves an angle of imaging view) from the captured image in eachimaging frame, or a feature point having a movement amount with the samedegree as other detected feature points is determined as a feature pointother than the main subject. This is a method in which a difference ofmovement in the captured image between the main subject for which aphotographer has an aim that it should intentionally be within an angleof imaging view and other subjects that are unintentionally moved due tohand shake or the like is used for determination. Feature pointcoordinates belonging to the partitioned main subject area are retainedin a predetermined storage area in the memory (S114). Feature pointcoordinates belonging to areas other than the main subject, for example,a background area, are retained in a predetermined storage area in thememory (S115). Next, the process advances to S106.

Next, a position and attitude estimation process will be described withreference to FIG. 2 and S101 to S107 in FIG. 5.

The position and attitude estimation unit 207 acquires a shake detectionsignal from the first vibration sensor 201 (S101). A differentiator 217in FIG. 2 performs a differential operation on an output of the A/Dconverter 218 and outputs the computation result to a subtractor 216.When a difference of position detection signals between imaging framesof the correction lens 103 is calculated, a movement speed of thecorrection lens 103 is calculated (S102).

The subtractor 216 acquires outputs of the A/D converter 202 and thedifferentiator 217, subtracts a movement speed of the correction lens103 from angular velocity detection information according to the firstvibration sensor 201, and thereby calculates information correspondingto a shake correction remaining angular velocity of the image pick-upapparatus 100 (S103). The output of the subtractor 216 is input to theposition and attitude estimation unit 207 in FIG. 2. The position andattitude estimation unit 207 acquires acceleration information appliedto the image pick-up apparatus 100 detected from the A/D converter 204by the second vibration sensor 203 (S104). The position and attitudeestimation unit 207 estimates a position and attitude of the imagepick-up apparatus 100 in the real space (S105). The feature coordinatemap estimation unit 206 estimates 3D position coordinates including adepth of a feature point in the real space for the image pick-upapparatus and generates a feature coordinate map (S106). The featurecoordinate map is a map of 3D position coordinates estimated on thebasis of the estimated position and attitude information of the imagepick-up apparatus 100 and coordinate change information of frames of themoving image according to 2D feature points in the captured image otherthan the main subject calculated by the main subject separation unit208.

The position and attitude estimation unit 207 corrects position andattitude estimation values obtained in S105 based on feature coordinatemap information, the estimated position and attitude of the imagepick-up apparatus 100, and 2D feature point coordinates in the capturedimage other than the main subject calculated by the main subjectseparation unit 208 (S107). When the process of estimating a positionand attitude and the process of estimating a feature coordinate map arerepeatedly performed for frames of the moving image, it is possible tocorrectly estimate a position and attitude. Here, position and attitudeestimation information in S105 is calculated from coordinates of featurepoints calculated from the image when shake correction is performed bythe correction lens 103 and a shake correction remaining angle obtainedby subtracting a movement speed of the correction lens 103 from angularvelocity detection information according to the first vibration sensor201. The position and attitude estimation values and feature coordinatemap information according to the estimation unit 205 are output to thecomputation unit 219.

Next, a process of computing a main subject feedback amount performed bythe computation unit 219 will be described with reference to FIG. 3B andFIG. 6. FIG. 3B is a diagram showing a detailed internal configurationof the computation unit 219. The computation unit 219 acquires globalvector information calculated by the global vector computation unit 220.An integrator 305 integrates global vector information and calculates anamount of movement of pixels in the captured image. A conversion unit308 acquires the position and attitude estimation values and 3D spacefeature coordinate map information including depth information estimatedby the estimation unit 205 and converts the 3D space feature coordinatemap information into feature point coordinates in the captured image.

A first subtractor 309 subtracts an output of the main subjectseparation unit 208 from an output of the conversion unit 308. Theoutput of the main subject separation unit 208 corresponds tocoordinates of feature points of the main subject. The first subtractor309 outputs the signal after subtraction to a second subtractor 306. Thesecond subtractor 306 subtracts an output of the first subtractor 309from an output of the integrator 305 and outputs it to an angleconversion gain unit 307. The angle conversion gain unit 307 multipliesby a gain value in order to convert the calculated pixel movement amountinto a value corresponding to an angle, and outputs the calculatedcontrol amount to the target position calculation unit 213. Thus it willbe understood that the computation unit 219 may calculate a controlamount of shake correction.

FIG. 6 is a flowchart showing a main subject feedback amount computingprocess. The processes of S121 and S122 and the processes of S125 toS129 are performed as parallel processes. In S121, the computation unit219 acquires a global vector calculated by the global vector computationunit 220. Next, the integrator 305 integrates acquired global vectorvalues and calculates a pixel movement amount (S122).

On the other hand, the computation unit 219 acquires the position andattitude estimation values of the image pick-up apparatus 100 estimatedby the estimation unit 205 (S125). The computation unit 219 acquires a3D feature coordinate map including a depth other than the main subjectestimated from feature points belonging to an area other than the mainsubject by the main subject separation unit 208 (S126). The conversionunit 308 converts the 3D feature coordinate map including a depth otherthan the main subject into 2D feature coordinates in the captured imageusing feature point coordinates and the position and attitude estimationvalues of the image pick-up apparatus (S127). First, a process ofconverting 3D feature coordinates in a world coordinate system of anobject other than the main subject into 3D feature coordinates in acamera coordinate system is performed. The world coordinate system is afixed coordinate system that defines coordinates of an object regardlessof a position of a camera. Details will be described with reference toFIG. 7.

FIG. 7 is a diagram showing the relationship between a coordinateposition of an object in world coordinates and a coordinate position incamera coordinates. T represents a vector from a starting point OW inworld coordinates to a starting point OC in camera coordinates. (rx, ry,rz) represents a unit vector indicating directions of axes (x, y, z) incamera coordinates when viewed in world coordinates. A point (x, y, z)in a camera coordinate system is represented as a point (X, Y, Z) in theworld coordinate system. The relationship between these coordinates isas shown in the following Formula 1.

$\begin{matrix}{\begin{bmatrix}x \\y \\z\end{bmatrix} = {{{R\left( {\begin{bmatrix}X \\Y \\Z\end{bmatrix} - T} \right)}\mspace{14mu} R} = \begin{bmatrix}r_{x}^{T} \\r_{y}^{T} \\r_{z}^{T}\end{bmatrix}}} & {{Formula}\mspace{14mu} 1}\end{matrix}$

In Formula 1, R represents a rotation matrix, and T represents aparallel movement vector.

Next, conversion from 3D feature coordinates of the camera coordinatesystem into image coordinates is performed by, for example, perspectiveconversion. FIG. 8 shows a perspective projection model when a virtualimaging plane is set at a position of a focus distance f in front of alens. The point O in FIG. 8 represents the center of a camera lens and Zaxis represents an optical axis of a camera. In addition, a coordinatesystem including the point O as a starting point is called the cameracoordinate system. (X, Y, Z) represents a coordinate position of anobject in the camera coordinate system. Image coordinates projected fromthe camera coordinates (X, Y, Z) of an object according to perspectiveconversion are represented as (x, y). The formula of conversion from (X,Y, Z) to (x, y) is represented by the following Formula 2.

$\begin{matrix}{{x = {f\frac{X}{Z}}},{y = {f\frac{Y}{Z}}}} & {{Formula}\mspace{14mu} 2}\end{matrix}$

In this manner, a 3D feature coordinate map including a depth other thanthe main subject can be converted into 2D feature coordinates in thecaptured image.

In S128 in FIG. 6, feature coordinates belonging to a main subject areaseparated by the main subject separation unit 208 are acquired. Next,the first subtractor 309 compares movements in moving image frames offeature coordinates of the main subject area acquired in S128 andfeature coordinates outside the main subject area estimated in S127. Adifference amount is calculated by subtraction processing (S129). Thesecond subtractor 306 subtracts a difference amount calculated in S129from a movement amount of the global vector calculated in S122 (S123).The angle conversion gain unit 307 calculates a main subject feedbackamount by multiplying by a gain value so that a value corresponding toan angle is obtained, and outputs the result to the target positioncalculation unit 213 (S124). The main subject feedback amount isrepresented by the following Formula 3.Main subject feedback amount=global vector movement amount−(featurecoordinate movement amount of background area−feature coordinatemovement amount of main subject)  Formula 3

In the target position calculation unit 213, the adder 304 adds a mainsubject feedback amount to a target position of the correction lens 103calculated on the basis of a detection output of the first vibrationsensor 201 (refer to FIG. 3A).

With reference to FIG. 9A and FIG. 9B, and FIG. 10A and FIG. 10B, aneffect when a target position to which a main subject feedback amount isadded is used for shake correction control will be described. FIG. 9Aand FIG. 9B are diagrams showing the positional relationship betweencamera work (position and attitude change) of an image pick-up apparatusand a subject. FIG. 10A and FIG. 10B are diagrams showing therelationship of feature point coordinates in the captured image duringimaging.

FIG. 9A is a schematic diagram showing position and attituderelationships between a main subject 504, background subjects 502 and503 which are on the back side close to the main subject, and an imagepick-up apparatus. A state when a user changes camera work while themain subject 504 is captured at the center of an angle of view when amoving image is captured is shown. A first moving image capturing frame901 is denoted as a frame 1 and the next second moving image capturingframe 902 is denoted as a frame 2. The example of FIG. 9A shows camerawork in which a position and attitude of the image pick-up apparatus arechanged.

FIG. 10A schematically shows a positional relationship between featurepoint coordinates of the background subjects 502 and 503 and the mainsubject 504 in a captured image 501 during camera work imaging from theframe 1 to the frame 2. FIG. 10A shows camera work imaging in which themain subject 504 remains at the center of an imaging screen. Regardingthe positional change of feature coordinates from the frame 1 to theframe 2, a movement amount of feature coordinates of the main subject isrelatively smaller than a movement amount of feature coordinates of thebackground. Coordinates change so that the image of the main subject 504remains in that place. Therefore, in Formula 3, a global vector movementamount which is the most frequent uniform movement in the entire imagingscreen and a feature coordinate movement amount of the background areaare in the same movement. The feature coordinate movement amount of thebackground area in this case is a movement amount obtained by converting3D feature coordinates including a depth of the background subject 502or 503 into 2D feature coordinates by the method based on position andattitude estimation values of the image pick-up apparatus. Since theglobal vector movement amount and the feature coordinate movement amountof the background area are in a relationship in which they are canceledout, the main subject feedback amount is equal to the feature coordinatemovement amount of the main subject. According to the main subjectfeedback amount, the correction lens 103 is controlled so that amovement amount of feature coordinate values of the main subject becomeszero. That is, correction is performed so that a change in coordinatevalues of the main subject on the imaging screen becomes smaller.Control is performed so that a change in the angle of view due to thecamera work causing a movement of the entire imaging screen is notreduced and only a movement of the main subject on the imaging screen isreduced.

FIG. 9B is a schematic diagram showing camera work when only an attitudeof the image pick-up apparatus is changed regardless of the position ofthe main subject 504 in position and attitude relationships between themain subject 504, the background subjects 502 and 503, and the imagepick-up apparatus. During change from the frame 1 to the frame 2, anattitude and an imaging direction of the image pick-up apparatus change.FIG. 10B shows a positional relationship between feature pointcoordinates of the background subjects 502 and 503 and the main subject504 in the captured image 501 when camera work is captured. In suchcamera work, imaging not intended by a photographer who aims tointentionally put the main subject in the screen is performed. A changein the angle of imaging view is a change in an angle of view that isdesired to be reduced by a photographer, for example, a change in theangle of view due to hand shake or the like. In this case, the featurecoordinate movement amount of the background area corresponding to thebackground subject 502 or 503 and the feature coordinate movement amountof the main subject 504 are in the same movement, and are in arelationship in which they are canceled out. Therefore, the main subjectfeedback amount is equal to the global vector movement amount accordingto Formula 3. According to the main subject feedback amount, since thecorrection lens 103 is controlled so that the most frequent uniformmovement amount in the entire imaging screen becomes zero, shakecorrection is performed so that a change in the angle of view of theentire imaging screen becomes smaller. That is, image blur due to shakeof the image pick-up apparatus such as hand shake which causes a changein the angle of view of the entire imaging screen is corrected.

In the present embodiment, when the main subject feedback amount iscalculated according to Formula 3, it is possible to determine which ofa movement between a movement of the entire imaging screen and amovement of main subject coordinates on the imaging screen is to bereduced without performing a complex determination process.

In a subject tracking operation of the related art, since a movement ofa subject is simply determined on a captured image, and correction isperformed so that a position of a subject form remains at a specificposition, it is not possible to determine whether a movement of thesubject on the captured image is caused due to the movement of thesubject or due to a movement of the image pick-up apparatus. Inaddition, in a method of the related art, a magnitude of the movementvisible in the captured image due to a difference of depth distancesbetween the subject and the image pick-up apparatus changes, and it isnot possible to ascertain an actual movement amount of each subject. Onthe other hand, in the present embodiment, regarding relationshipsbetween feature coordinates of the main subject and the background andthe position of the image pick-up apparatus, 3D coordinates including adepth are determined and movements of the subject and the image pick-upapparatus are determined, and thus it is possible to appropriatelydetermine and control respective movements.

In addition, in the present embodiment, using the position and attitudeof the image pick-up apparatus estimated in S127 in FIG. 6 and featurecoordinates on the captured image estimated from 3D feature coordinatesbelonging to the background area, it is possible to estimate a movementof the feature points other than the main subject (such as thebackground). If feature points of the background area tracked on thecaptured image are moved outside the imaging screen by camera work, itis possible to continue to estimate feature coordinates of thebackground area. Alternatively, even if feature points are hidden behindanother object or even if tracking is not possible due to a change inimaging situation such as a luminance change of the captured image, itis possible to continue to estimate feature coordinates of thebackground area.

In the present embodiment, in imaging according to camera work for asubject tracking operation, image blur correction is performed byseparating a change in the position and attitude of the image pick-upapparatus due to hand shake and a change in the position and attitudedue to the camera work. Therefore, it is possible to obtain a favorablecaptured image in which an unnatural change in the angle of viewoccurring in a shake correction method of the related art is curbed.

Second Embodiment

Next, a second embodiment of the present invention will be described.The present embodiment is different from the first embodiment in thatelectronic image blur correction is performed by image processing. Inthe present embodiment, the same reference numerals are used for partsthe same as in the first embodiment and details thereof will be omitted.FIG. 11 is a diagram showing a configuration example of the image blurcorrection device of the present embodiment. Differences from those ofthe configuration example shown in FIG. 2 will be described below. Inthe image blur correction device of the present embodiment, thecomputation unit 219 outputs the calculated main subject feedback amountto the imaging signal processing unit 110.

In the present embodiment, there is no process of adding a main subjectfeedback amount to the target position of the correction lens 103 shownin S120 in FIG. 4 in the first embodiment. Alternatively, a process forinstructing the imaging signal processing unit 110 regarding acoordinate position at which an image signal is read is performed by thecomputation unit 219 based on the main subject feedback amount. Theimaging signal processing unit 110 performs image blur correction bychanging a position at which an image signal after imaging is extracted.

In the present embodiment, it is possible to realize an electronic imageblur correction process of changing a coordinate position at which animage signal output from the imaging unit 109 is read. The optical imageblur correction process and the electronic image blur correction processcan be used together or switched between according to imagingconditions, a shake state, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-150810, filed Aug. 3, 2017, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. An image pick-up apparatus that acquires an imagesignal by an imaging unit, comprising: a memory; and one or moreprocessors, wherein the processor functions as the following unitsaccording to a program stored in the memory: a first acquisition unitconfigured to acquire first information indicating shake of the imagepick-up apparatus detected by a shake detection unit: a secondacquisition unit configured to acquire second information indicating amovement of a subject detected in an image signal by the imaging unit; atracking unit configured to calculate coordinate values of the subjecton an imaging screen using the second information and track featurepoints; an estimation unit configured to estimate a position and/orattitude of the image pick-up apparatus and a positional relationshipincluding a depth between the subject and the image pick-up apparatusfrom the first information and the coordinate values of the subject; acomputation unit configured to calculate a control amount of shakecorrection using (i) the estimation value of the position or attitude ofthe image pick-up apparatus acquired from the estimation unit, (ii) thepositional relationship acquired from the estimation unit, (iii) thefirst information and (iv) the calculated coordinate values of thesubject; and a correction unit configured to correct image blur due toshake of the image pick-up apparatus based on the control amountcalculated by the computation unit.
 2. The image pick-up apparatusaccording to claim 1, wherein the computation unit calculates thecontrol amount corresponding to a change in the position or attitude ofthe image pick-up apparatus according to camera work of tracking thesubject.
 3. The image pick-up apparatus according to claim 1, whereinthe shake detection unit is an angular velocity sensor and/or anacceleration sensor.
 4. The image pick-up apparatus according to claim1, wherein the processor further function as a separation unitconfigured to acquire an output of the tracking unit and separatefeature points of a first subject and feature points of a secondsubject, wherein the separation unit outputs coordinates of the featurepoints of the second subject to the estimation unit.
 5. The imagepick-up apparatus according to claim 4, wherein the estimation unitgenerates feature coordinate map information related to 3D positioncoordinates in a real space from the first information and coordinatechange information related to the feature points of the second subjectcalculated by the separation unit, and calculates the estimation valueof the position or attitude of the image pick-up apparatus using thefeature coordinate map information and the coordinates of the featurepoints of the second subject calculated by the separation unit.
 6. Theimage pick-up apparatus according to claim 4, wherein the processorfurther functions as a calculation unit configured to acquire the secondinformation and calculate a global vector representing a movement of theentire imaging screen, wherein the computation unit computes adifference between a movement amount of the feature points of the secondsubject acquired from the estimation unit and a movement amount of thefeature points of the first subject acquired from the separation unit,and calculates a feedback amount by subtracting the difference from theglobal vector and outputs the feedback amount to the correction unit. 7.The image pick-up apparatus according to claim 5, wherein the firstsubject is a main subject and the second subject is a background, andwherein the estimation unit estimates a movement of the feature pointsof the second subject using the position or attitude of the imagepick-up apparatus and feature coordinates on a captured image estimatedfrom 3D feature coordinates belonging to an area of the second subject.8. The image pick-up apparatus according to claim 5, wherein thecomputation unit includes: a conversion unit configured to (i) acquirethe estimation value of the position or attitude of the image pick-upapparatus and the feature coordinate map information from the estimationunit and (ii) convert the feature coordinate map information intofeature coordinates on an imaging screen; and a subtraction unitconfigured to subtract a movement amount of the feature points of thefirst subject from an output of the conversion unit.
 9. The imagepick-up apparatus according to claim 8, wherein the computation unit andthe correction unit perform control such that a change in the coordinatevalues of the first subject in the imaging screen becomes smaller whenthe movement amount of the feature points of the first subject issmaller than a movement amount of the feature points of the secondsubject.
 10. The image pick-up apparatus according to claim 8, whereinthe computation unit and the correction unit perform control such that achange in the entire imaging screen due to shake becomes smaller when amovement amount of the feature points of the second subject and themovement amount of the feature points of the first subject are the same.11. The image pick-up apparatus according to claim 1, wherein theprocessor further functions as: a separation unit configured to acquirean output of the tracking unit and separate feature points of a firstsubject and feature points of a second subject, wherein the separationunit outputs coordinates of the feature points of the second subject tothe estimation unit, and wherein the computation unit calculates thecontrol amount corresponding to a change in the position or attitude ofthe image pick-up apparatus according to camera work of tracking thesubject.
 12. The image pick-up apparatus according to claim 11, whereinthe estimation unit generates feature coordinate map information of 3Dposition coordinates from the first information and coordinate changeinformation related to the feature points of the second subjectcalculated by the separation unit, and calculates the estimation valueof the position or attitude of the image pick-up apparatus using thefeature coordinate map information and the coordinates of the featurepoints of the second subject calculated by the separation unit.
 13. Theimage pick-up apparatus according to claim 12, wherein the first subjectis a main subject and the second subject is a background, and whereinthe estimation unit estimates a movement of the feature points of thesecond subject using the position or attitude of the image pick-upapparatus and feature coordinates on a captured image estimated from 3Dfeature coordinates belonging to an area of the second subject.
 14. Theimage pick-up apparatus according to claim 12, wherein the computationunit includes a conversion unit configured to acquire the estimationvalue of the position or attitude of the image pick-up apparatus and thefeature coordinate map information from the estimation unit and convertthe feature coordinate map information into feature coordinates on animaging screen and a subtraction unit configured to subtract a movementamount of the feature points of the first subject from an output of theconversion unit.
 15. The image pick-up apparatus according to claim 14,wherein the computation unit and the correction unit perform controlsuch that a change in the coordinate values of the first subject in theimaging screen becomes smaller when the movement amount of the featurepoints of the first subject is smaller than a movement amount of thefeature points of the second subject.
 16. The image pick-up apparatusaccording to claim 14, wherein the computation unit and the correctionunit perform control such that a change in the entire imaging screen dueto shake becomes smaller when a movement amount of the feature points ofthe second subject and the movement amount of the feature points of thefirst subject are the same.
 17. A method executed in an image pick-upapparatus that acquires an image signal by an imaging unit, the methodcomprising: acquiring first information indicating shake of the imagepick-up apparatus detected by a shake detection unit and secondinformation indicating a movement of a subject detected in an imagesignal by the imaging unit; calculating coordinate values of the subjecton an imaging screen using the second information and tracking featurepoints; estimating a position and/or attitude of the image pick-upapparatus and a positional relationship including a depth between thesubject and the image pick-up apparatus from the first information andthe coordinate values of the subject; calculating a control amount ofshake correction using (i) the estimated position or attitude of theimage pick-up apparatus, (ii) the positional relationship, (iii) thefirst information, and (iv) the calculated coordinate values of thesubject; and correcting image blur due to shake of the image pick-upapparatus based on the control amount calculated in the calculating.